feat(client): retry transient GET failures with backoff#37
Conversation
| response: requests.Response | None = None | ||
| for attempt in range(self._MAX_RATE_LIMIT_RETRIES + 1): | ||
| response = self._request_with_same_origin_redirects( | ||
| response = self._request_with_transient_retries( |
There was a problem hiding this comment.
(AI-assisted)
Could we enforce the advertised seven-attempt budget across the composed retry layers?
429 is correctly excluded from _RETRYABLE_GET_STATUSES, but each outer rate-limit retry calls _request_with_transient_retries() again with a fresh budget. A sequence of six 503s followed by 429, repeated across the four rate-limit rounds, therefore makes 28 physical requests for one logical GET; the 401 refresh path can repeat the composition again.
Could we use one shared attempt/deadline budget across transient, rate-limit, and auth-refresh retries, and add a mixed 503/429 regression test that asserts the total request count? That would avoid retry amplification during an outage while preserving Retry-After handling.
429fe5a to
a575da1
Compare
Idempotent GETs now retry on 500/502/503/504 and transient transport errors (connection reset/refused, timeout, truncated body) with doubling backoff capped at 30s, up to 7 attempts per logical request. Mutating methods are never duplicated and streamed GETs are excluded so stream consumers keep control of the open path. SSL certificate errors raise on the first attempt because they are deterministic and retrying only delays the report. The existing 429 Retry-After handling is unchanged and composes with the new layer. Retry warnings go through the client logger; pipeline-run, artifact, pipeline hydration, published-component, and secret commands thread their --log-type logger into the client so the warnings follow the configured sink, and logger-less programmatic clients stay silent.
a575da1 to
3f3421b
Compare
2 participants
Summary
requests.exceptions.SSLErroron the first attempt: certificate failures are deterministic, so retrying only delays the report--log-typelogger to the client, while logger-less programmatic clients remain silentContext
Short backend restarts and connection resets currently fail read-only CLI operations immediately even though repeating the request is safe. This adds bounded recovery for GET requests without risking duplicate writes or changing rate-limit behavior, and keeps every retry visible through the logger each CLI command already configures.
Testing
uv run pytest tests/test_client.py tests/test_pipeline_runs_cli.py tests/test_artifacts_cli.py tests/test_components_cli.py tests/test_secrets_cli.pyuvx ruff checkon the touched source filesuv lock --checkgit diff --check